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ABSTRACT 



Technology and testing have become two popular prescriptions 
for improving education. The technology nostrum suggests that the infusion of 
modern technology into schools will bolster teaching and learning and will 
prepare students for an increasingly technological workplace. The testing 
prescription holds that using standardized test scores to rate schools and to 
decide whether students should be promoted or graduate will provide 
incentives for improvement. What is not often recognized is that these two 
prescriptions may work against each other. Research findings suggest a great 
gap between computer use in schools and testing strategies used for school 
improvement, a gap that will increase as more students become more accustomed 
to writing on computers. There are at least three possible ways to reduce 
this gap: (1) decrease students' computer time so that they do not become 

accustomed to writing on computers, retaining the paper-and-pencil skills 
needed for current kinds of tests; (2) have students complete their tests on 
a computer; and (3) perhaps the most reasonable solution is to recognize the 
limitations of current testing programs, and acknowledge that the scores of 
high-stakes tests do not measure the capabilities of some students 
adequately. (Contains 12 endnotes.) ( SLD) 
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In 1983, the release of A Nation at Risk by the US Department of Education 
focused attention on the perceived crisis in education. Since then, technol- 
ogy and testing have become two popular prescriptions for improving 
education. 

The technology nostrum is the infusion of modem technology into 
schools, in the belief that it will bolster teaching and learning and prepare 
students for an increasingly technological workplace. The testing prescrip- 
tion holds that using standardized test scores to rate schools and to decide 
whether students should be promoted or graduate will provide incentives 
for improvement. What is little recognized is that these two prescriptions 
may work against each other. Recent research shows that standardized lan- 
guage arts tests taken on paper severely underestimate the performance of 
students accustomed to working on computer. 1 It is like asking mathemati- 
cians to abandon calculators and revert to slide rules. 
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The Computer Revolution Goes to School 

Though the personal-computer revolution began only 
twenty years ago and the World Wide Web is even newer, com- 
puter technology has already had a dramatic impact on soci- 
ety. Schools have been slower to acquire these technologies, but 
computer use in schools is increasing rapidly. 2 The percentage 
of students in grades 1 to 8 using computers in school has more 
than doubled, from 31.5 in 1984 to 68.9 in 1993. 3 Similarly, while 
schools had one computer for every 125 students in 1983, they 
had one for every 9 in 1995. 4 

And not only are there more computers in classrooms, 
schools are also increasing students' use of computers and ac- 
cess to the Internet. A recent national survey of teachers showed 
that in 1998, 50 percent of K-12 teachers had students use word 
processors, 36 percent had them use CD ROMS, and 29 percent 
had them use the World Wide Web. 5 In short, the computer 
revolution has gone to school, and more and more students are 
writing and doing school assignments and research on computers. 



Performance Testing in Schools 

Meanwhile, many states are increasingly seeking to hold 
students, teachers and schools "accountable" for student learn- 
ing as measured by state-sponsored tests. According to annual 
surveys by the Council for Chief State School Officers (1998), 
48 states use statewide tests to assess student performance in 
different subject areas. 6 Because of the limitations of multiple- 
choice items, most statewide tests include items for which stu- 
dents must write extended answers or explain their work. Last 
year alone, an estimated ten million students nationwide par- 
ticipated in a state-sponsored testing program that required 
them to write responses longhand. Scores on these tests are 
being used to determine whether to (1) promote students to 
higher grades, (2) grant high school diplomas, and (3) identify, 
sanction or reward low- and high-performing schools. 

We wish to focus here on a little-recognized limitation of 
using these tests to drive educational "reform" - the fact that 
paper-and-pencil forms of these tests may yield misleading in- 
formation on the capabilities of students who are accustomed 
to using computers. 
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Testing Via Computer 



Research on testing via computer goes back several decades 
and suggests that for multiple-choice tests, administration via 
computer yields about the same results as via paper and pen- 
cil. 7 However, more recent research shows that for young people 
who have gone to school with computers, national and state 
tests administered via paper and pencil can yield severe un- 
derestimates of students' skills as compared with the same tests 
administered via computer. 8 

This research began with a puzzle. While evaluating the 
progress of student learning in the Accelerated Learning 
Laboratory (ALL), a high-tech school in Worcester, MA, teach- 
ers were surprised by the results from the second year of as- 
sessments. Although their students were writing more often 
now that computers were in the school, their scores on writing 
tests declined. To help solve the puzzle, it was decided to com- 
pare paper and computer administration of the tests. 

In 1995, a randomized experiment was conducted, with one 
group of students taking math, science and language arts tests, 
including both multiple-choice and open-ended items, on pa- 
per, and another group taking the tests on computer. Before 
scoring, answers written by hand were transcribed so that rat- 
ers could not distinguish them from those done on computer. 
There were two major findings. First, the multiple-choice test 
results did not differ much by mode of administration. But sec- 
ond, for the ALL students used to writing on the computer, 
responses written on computer were much better than those 
written by hand. This finding occurred across all three subjects 
and with both short-answer and extended-answer items. The 
effects were so large that when students wrote on paper, only 
30 percent performed at a "passing" level; when they wrote on 
computer, 67 percent "passed." 9 

Two years later, a more sophisticated study was conducted, 
this time using open-ended items from the new Massachusetts 
state test (the Massachusetts Comprehensive Assessment System 
or MCAS) and the National Assessment of Educational Progress 
(NAEP) in the areas of language arts, science and math. Again, 
eighth grade students from two middle schools in Worcester, 
MA, were randomly assigned to groups. Within each subject 
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area, each group was given the same test items, with one group 
answering on paper and the other on computer. In addition, 
data on students' keyboarding speed and prior computer use 
were collected. Finally, all answers written by hand were tran- 
scribed to computer text. 

As in the first study, large differences were evident on the 
language arts tests. For students who could keyboard moder- 
ately well (20 words per minute or more), performance on com- 
puter was much better than on paper. Overall, the difference 
represented more progress than the average student makes in 
an entire year and could raise a student's score on MCAS from 
the "needs improvement" to the "passing" level. 10 



Figure 1 



Effect of Computer Test Administration on Language Arts Test 

by Level of Typing Ability 
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Recalling that nearly ten million students took some type 
of state-sponsored written test last year and that nearly half of 
the students nationwide use word processors in school, these 
results suggest that state paper-and-pencil tests may be under- 
estimating the abilities of some five million students annually. 

Study findings were not, however, consistent across all lev- 
els of keyboarding proficiency (see Figure 1). As keyboarding 
speed decreased, the benefit of computer administration be- 
came smaller. And at very low keyboarding speed, taking the 
test on computer diminished students' performance. Similarly, 
taking the math test on computer had a negative effect on stu- 
dents' scores, which became less pronounced as keyboarding 
speed increased. 



◄ 



Bridging the Gap 

These studies highlight a huge gap between computer use 
in schools and testing strategies used for school improvement 
- one that will increase as more students become accustomed 
to writing on computers. There are at least three possible ways 
to bridge this gap. 
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First, schools can decrease students' computer time so that 
they do not become accustomed to writing on computers. Some 
schools have already adopted this practice. After the first study 
described above, and the introduction of the new paper-and- 
pencil MCAS test in Massachusetts, the ALL school required 
students to write more on paper and less on computer. 11 In an- 
other Massachusetts school system, the principal feared that 
students who write regularly on computer lose penmanship 
skills, which might lead to lower scores on the new state test. 
This school increased penmanship instruction across all grades 
while also decreasing students' time on computers. 12 Such prac- 
tices - in effect de-emphasizing computers in schools to better 
prepare students for low-tech tests - may be pragmatic, given 
the high stakes attached to many state tests. But they may be 
shortsighted in light of students' entry into an increasingly high- 
tech world. 
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The full reports 
referenced in this 
paper are available 
on the World Wide 
Web. Go to http:// 
epaa.asu.edu. 

Then go to volume 5, 
number 3 and 
volume 7 no. 20. 



A second way to bridge the test-technology gap would be 
to eliminate paper-and-pencil testing and have students com- 
plete tests on computer. This might seem a sensible solution, 
but it will not be feasible until our schools obtain an adequate 
technology infrastructure. Moreover, as shown by problems in 
recent moves to computer-administer some large-scale tests for 
adults, computerized testing is not the panacea some had 
hoped. Among other problems, it adds considerably to the cost 
of testing and creates new test security concerns. Finally, as our 
second study showed, it would penalize low-tech students with 
poor keyboard skills. 

A third approach, and perhaps the most reasonable solu- 
tion in the short term, is to recognize the limitations of current 
testing programs. Without question, both computer technol- 
ogy and performance testing can help improve the quality of 
education. However, until students can take tests in the same 
medium in which they generally work and learn, we must rec- 
ognize that the scores from high-stakes state tests do not accu- 
rately measure some students' capabilities. While this does not 
make the scores useless, it serves as yet another reminder of 
the dangers of making decisions based solely on test scores. ;V 
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About the National Board 
on Educational Testing 
and Public Policy 

Created as an independent monitoring system 
for assessment in America, the National Board on 
Educational Testing and Public Policy is located in 
the Peter S. and Carolyn A. Lynch School of 
Education at Boston College. The National Board 
provides research-based test information for policy 
decision making, with special attention to groups 
historically underserved by the educational 
systems of our country. Specifically, the 
National Board 

• Monitors testing programs, policies, and 
products 

• Evaluates the benefits and costs of testing 
programs in operation 

• Assesses the extent to which professional 
standards for test development and use are 
met in practice 
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